Surgical robotic system having grip-dependent control

ABSTRACT

Surgical robotic systems, and methods of controlling such systems based on a user&#39;s grip on a user interface device, are described. During a surgical procedure, a surgical robotic system generates control commands to actuate a component of the surgical robotic system. The component can be a surgical tool that is actuated to move based on tracking data from the user interface device. The component can be the user interface device, which is actuated to render haptic feedback to the user&#39;s hand based on load data from a surgical robotic arm holding the surgical tool. In either case, the actuation is based on a center of rotation of the user interface device that corresponds to the user&#39;s grip on the user interface device. Other embodiments are described and claimed.

BACKGROUND Field

Embodiments related to robotic systems are disclosed. More particularly,embodiments related to surgical robotic systems having user interfacedevices are disclosed.

Background Information

Endoscopic surgery involves looking into a patient's body and performingsurgery inside the body using endoscopes and other surgical tools. Forexample, laparoscopic surgery can use a laparoscope to access and viewan abdominal cavity. Endoscopic surgery can be performed using manualtools and/or a surgical robotic system having robotically-assistedtools.

A surgical robotic system may be remotely operated by a surgeon tocommand robotically-assisted tools or a camera located at an operatingtable. The surgeon may use a user console located in the operating room(or in a different city) to command a robot to manipulate the surgicaltool and the camera. For example, the surgeon may hold in her hand auser input device such as a joystick or a computer mouse that shemanipulates to generate control commands that cause motion of thesurgical robotic system components, e.g., the surgical tool or thecamera. The robot can use the surgical tools to perform surgery, withthe visualization aid provided by the camera.

Accurate control of robotically-assisted tools is important to reachinga favorable clinical outcome with robotic surgery. Such controltypically requires that an input movement of the user input device beaccurately translated to a corresponding movement of a roboticallyassisted tool.

SUMMARY

User input devices used to generate control commands to cause motion ofsurgical robotic system components may be ungrounded. For example, auser input device can be held by the surgeon and freely manipulated inspace without being mechanically linked to another structure in theoperating room. In such case, a center of rotation of the user inputdevice can be mapped to a center of rotation of the surgical tool.Accordingly, when the surgeon moves, e.g., tilts or rotates, the userinput device, a corresponding movement of the surgical tool can occur.The center of rotation can be fixed, e.g., to coincide with a trackingsensor in the user interface device used to track movement of the userinterface device. The fixed center of rotation, however, may not matchthe expectation of the surgeon. For example, when the surgeon grips theuser interface device in front of or behind the fixed center ofrotation, the surgeon may expect the center of rotation to be betweenher fingers at a location that is offset from the fixed center ofrotation. As a result, the surgeon may move the user input deviceexpecting a corresponding movement of the surgical tool, but thesurgical tool may move in a different manner that does not accuratelyemulate the expected movement of the surgeon. Accordingly, a surgicalrobotic system that determines a virtual center of rotation matching theexpectation of the user, and that uses the virtual center of rotation tocontrol the surgical tool, can contribute to accurate control of thesurgical robotic system and successful robotic surgery.

A surgical robotic system and a method of controlling such systems basedon a user's grip on a user interface device, are provided. In anembodiment, the method includes determining a grip configuration of auser's hand on the user interface device. The user interface device caninclude proximity sensors, such as time-of-flight sensors or capacitivesensors, which detect the positions of the fingers of the user's hand.The grip configuration can be defined based on the finger positions. Forexample, when the finger positions are near a distal end of the userinterface device, the grip configuration can be an anterior grip, andwhen the finger positions are near a proximal end of the user interfacedevice, the grip configuration can be a posterior grip.

The method includes determining, based on the grip configuration, acenter of rotation of the user interface device. The user interfacedevice can include a tracking sensor to detect movement of the userinterface device. More particularly, the movement may be detected abouta sensor center of rotation, e.g., tracking data can describe movementof the user interface device relative to the sensor center of rotation.By contrast, the virtual center of rotation determined based on the gripconfiguration may be offset from sensor center of rotation. The virtualcenter of rotation can be set to a predetermined location based onwhether the grip is anterior to or posterior to the sensor center ofrotation. The virtual center of rotation can be set based on geometricalanalysis of the positions of the user's fingers. In any case, thevirtual center of rotation can be an estimation of the center ofrotation expected by the user.

The surgical robotic system can control movement of a component based onthe determined virtual center of rotation of the user interface device.More particularly, the system can generate control commands to actuatethe component based on the virtual center of rotation. In an embodiment,the component is a surgical tool, and the control commands actuate thesurgical tool based on the virtual center of rotation and the trackingdata to move the surgical tool in six degrees of freedom. In anembodiment, the component is the user interface device, and the controlcommands actuate the user interface device based on the virtual centerof rotation and load data corresponding to a force applied to thesurgical tool to render haptic feedback to the user's hand emulating theforce. In any case, actuation of the component based on the virtualcenter of rotation expected by the user can provide accurate control ofthe component in a manner that matches the user's expectation.

The above summary does not include an exhaustive list of all aspects ofthe present invention. It is contemplated that the invention includesall systems and methods that can be practiced from all suitablecombinations of the various aspects summarized above, as well as thosedisclosed in the Detailed Description below and particularly pointed outin the claims filed with the application. Such combinations haveparticular advantages not specifically recited in the above summary.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the invention are illustrated by way of example andnot by way of limitation in the figures of the accompanying drawings inwhich like references indicate similar elements. It should be noted thatreferences to “an” or “one” embodiment of the invention in thisdisclosure are not necessarily to the same embodiment, and they mean atleast one. Also, in the interest of conciseness and reducing the totalnumber of figures, a given figure may be used to illustrate the featuresof more than one embodiment of the invention, and not all elements inthe figure may be required for a given embodiment.

FIG. 1 is a pictorial view of an example surgical robotic system in anoperating arena, in accordance with an embodiment.

FIG. 2 is a pictorial view of a surgical tool of a surgical roboticsystem, in accordance with an embodiment.

FIG. 3 is a pictorial view of a user interface device of a surgicalrobotic system, in accordance with an embodiment.

FIG. 4 is a pictorial view of a user interface device being held with afirst grip configuration, in accordance with an embodiment.

FIG. 5 is a pictorial view of a user interface device being held with asecond grip configuration, in accordance with an embodiment.

FIG. 6 is a flowchart of a method of actuating a surgical tool of asurgical robotic system based on a center of rotation of a userinterface device, in accordance with an embodiment.

FIG. 7 is a schematic view of a grip configuration on a user interfacedevice, in accordance with an embodiment.

FIG. 8 is a schematic view of a movement of a user interface deviceabout a center of rotation, in accordance with an embodiment.

FIG. 9 is a flowchart of a method of actuating a user interface deviceof a surgical robotic system based on a center of rotation of the userinterface device, in accordance with an embodiment.

FIG. 10 is a block diagram of exemplary hardware components of asurgical robotic system, in accordance with an embodiment.

DETAILED DESCRIPTION

Embodiments describe methods of controlling robotic systems based on auser's grip on a user interface device (UID). The UID can controlspatial motion of a surgical tool of a surgical robotic system during arobotic surgery. The UID may, however, be used in other robotic systems,such as for manufacturing or military applications, to name only a fewpossible applications.

In various embodiments, description is made with reference to thefigures. However, certain embodiments may be practiced without one ormore of these specific details, or in combination with other knownmethods and configurations. In the following description, numerousspecific details are set forth, such as specific configurations,dimensions, and processes, in order to provide a thorough understandingof the embodiments. In other instances, well-known processes andmanufacturing techniques have not been described in particular detail inorder to not unnecessarily obscure the description. Reference throughoutthis specification to “one embodiment,” “an embodiment,” or the like,means that a particular feature, structure, configuration, orcharacteristic described is included in at least one embodiment. Thus,the appearance of the phrase “one embodiment,” “an embodiment,” or thelike, in various places throughout this specification are notnecessarily referring to the same embodiment. Furthermore, theparticular features, structures, configurations, or characteristics maybe combined in any suitable manner in one or more embodiments.

The use of relative terms throughout the description may denote arelative position or direction. For example, “distal” may indicate afirst direction, e.g., along a longitudinal axis of a UID in a firstdirection. Similarly, “proximal” may indicate a second direction alongthe longitudinal axis opposite to the first direction. Such terms areprovided to establish relative frames of reference, however, and are notintended to limit the use or orientation of a surgical robotic systemcomponent to a specific configuration described in the variousembodiments below.

In an aspect, a surgical robotic system performs a method of controllingactuation of a system component based on a manner in which a user gripsa UID of the system. The grip configuration can be determined based ondetection of the user's finger location on the UID. For example,readings from proximity sensors, such as a time-of-flight (TOF) sensors,can be used to estimate the user's finger location on the UID. A controlsystem of the surgical robotic system can use the determined gripconfiguration to set a center of rotation of the UID. The center ofrotation of the UID can correspond to a center of rotation of a surgicaltool being controlled by the control system. Accordingly, movement ofthe UID about the UID center of rotation can be accurately translated tomovement of the surgical tool about the tool center of rotation. Thecenter of rotation of the UID can be adjusted in real time to maintainaccurate control of the surgical tool throughout the surgical procedure,even as the user changes her grip on the UID.

Referring to FIG. 1 , this is a pictorial view of an example surgicalrobotic system 1 in an operating arena. The system 1 includes a userconsole 2, a control tower 3, and one or more surgical robotic arms 4 ata surgical robotic platform 5, e.g., a table, a bed, etc. The arms 4 maybe mounted to a table or bed on which the patient rests as shown in theexample of FIG. 1 , or they may be mounted to a cart separate from thetable or bed. The system 1 can incorporate any number of devices, tools,or accessories used to perform surgery on a patient 6. For example, thesystem 1 may include one or more surgical tools 7 used to performsurgery. A surgical tool 7 may be an end effector that is attached to adistal end of a surgical arm 4, for executing a surgical procedure.

Each surgical tool 7 may be manipulated manually, robotically, or both,during the surgery. For example, the surgical tool 7 may be a tool usedto enter, view, or manipulate an internal anatomy of the patient 6. Inone aspect, the surgical tool 7 is a grasper that can grasp tissue ofthe patient. The surgical tool 7 may be configured to be controlledmanually by a bedside operator 8, robotically via actuated movement ofthe surgical robotic arm 4 to which it is attached, or both. The roboticarms 4 are shown as being table-mounted but in other configurations thearms 4 may be mounted to a cart, the ceiling or a sidewall, or toanother suitable structural support.

A remote operator 9, such as a surgeon or other human operator, may usethe user console 2 to remotely manipulate the arms 4 and their attachedsurgical tools 7, e.g., referred to here as teleoperation. The userconsole 2 may be located in the same operating room as the rest of thesystem 1 as shown in FIG. 1 . In other environments however, the userconsole 2 may be located in an adjacent or nearby room, or it may be ata remote location, e.g., in a different building, city, or country. Theuser console 2 may comprise a seat 10, foot-operated controls 13, one ormore handheld user input devices (UIDs) 14, and at least one userdisplay 15 that is configured to display, for example, a view of thesurgical site inside the patient 6. In the example user console 2, theremote operator 9 is sitting in the seat 10 and viewing the user display15 while manipulating a foot-operated control 13 and a handheld UID 14in order to remotely control the arms 4 and the surgical tools 7 thatare mounted on the distal ends of the arms 4.

In some variations, the bedside operator 8 may operate the system 1 inan “over the bed” mode in which the beside operator 8 (user) is at aside of the patient 6 and is simultaneously manipulating arobotically-driven tool (an end effector that is attached to the arm 4)with a handheld UID 14 held in one hand, and a manual laparoscopic toolin another hand. For example, the bedside operator's left hand may bemanipulating the handheld UID to control a robotically-driven tool,while the bedside operator's right hand may be manipulating a manuallaparoscopic tool. In this particular variation of the system 1, thebedside operator 8 can perform both robotic-assisted minimally invasivesurgery and manual laparoscopic surgery on the patient 6.

During an example procedure (surgery), the patient 6 is prepped anddraped in a sterile fashion to achieve anesthesia. Initial access to thesurgical site may be performed manually while the arms of the roboticsystem 1 are in a stowed configuration or withdrawn configuration (tofacilitate access to the surgical site.) Once access is completed,initial positioning or preparation of the robotic system 1 including itsarms 4 may be performed. Next, the surgery proceeds with the remoteoperator 9 at the user console 2 utilizing the foot-operated controls 13and the UIDs 14 to manipulate the various end effectors and perhaps animaging system, to perform the surgery. Manual assistance may also beprovided at the procedure bed or table, by sterile-gowned bedsidepersonnel, e.g., the bedside operator 8 who may perform tasks such asretracting tissues, performing manual repositioning, and tool exchangeupon one or more of the robotic arms 4. Non-sterile personnel may alsobe present to assist the remote operator 9 at the user console 2. Whenthe procedure or surgery is completed, the system 1 and the user console2 may be configured or set in a state to facilitate post-operativeprocedures such as cleaning or sterilization and healthcare record entryor printout via the user console 2.

In one embodiment, the remote operator 9 holds and moves the UID 14 toprovide an input command to move a robot arm actuator 17 in the roboticsystem 1. The UID 14 may be communicatively coupled to the rest of therobotic system 1, e.g., via a console computer system 16. The UID 14 cangenerate spatial state signals corresponding to movement of the UID 14,e.g. position and orientation of the handheld housing of the UID, andthe spatial state signals may be input signals to control a motion ofthe robot arm actuator 17. The robotic system 1 may use control signalsderived from the spatial state signals, to control proportional motionof the actuator 17. In one embodiment, a console processor of theconsole computer system 16 receives the spatial state signals andgenerates the corresponding control signals. Based on these controlsignals, which control how the actuator 17 is energized to move asegment or link of the arm 4, the movement of a corresponding surgicaltool that is attached to the arm may mimic the movement of the UID 14.Similarly, interaction between the remote operator 9 and the UID 14 cangenerate for example a grip control signal that causes a jaw of agrasper of the surgical tool 7 to close and grip the tissue of patient6.

The surgical robotic system 1 may include several UIDs 14, whererespective control signals are generated for each UID that control theactuators and the surgical tool (end effector) of a respective arm 4.For example, the remote operator 9 may move a first UID 14 to controlthe motion of an actuator 17 that is in a left robotic arm, where theactuator responds by moving linkages, gears, etc., in that arm 4.Similarly, movement of a second UID 14 by the remote operator 9 controlsthe motion of another actuator 17, which in turn moves other linkages,gears, etc., of the robotic system 1. The robotic system 1 may include aright arm 4 that is secured to the bed or table to the right side of thepatient, and a left arm 4 that is at the left side of the patient. Anactuator 17 may include one or more motors that are controlled so thatthey drive the rotation of a joint of the arm 4, to for example change,relative to the patient, an orientation of an endoscope or a grasper ofthe surgical tool 7 that is attached to that arm. Motion of severalactuators 17 in the same arm 4 can be controlled by the spatial statesignals generated from a particular UID 14. The UIDs 14 can also controlmotion of respective surgical tool graspers. For example, each UID 14can generate a respective grip signal to control motion of an actuator,e.g., a linear actuator, which opens or closes jaws of the grasper at adistal end of surgical tool 7 to grip tissue within patient 6.

In some aspects, the communication between the platform 5 and the userconsole 2 may be through a control tower 3, which may translate usercommands that are received from the user console 2 (and moreparticularly from the console computer system 16) into robotic controlcommands that transmitted to the arms 4 on the robotic platform 5. Thecontrol tower 3 may also transmit status and feedback from the platform5 back to the user console 2. The communication connections between therobotic platform 5, the user console 2, and the control tower 3 may bevia wired and/or wireless links, using any suitable ones of a variety ofdata communication protocols. Any wired connections may be optionallybuilt into the floor and/or walls or ceiling of the operating room. Therobotic system 1 may provide video output to one or more displays,including displays within the operating room as well as remote displaysthat are accessible via the Internet or other networks. The video output(video feed) may also be encrypted to ensure privacy and all or portionsof the video output may be saved to a server or electronic healthcarerecord system.

It will be appreciated that the operating room scene in FIG. 1 isillustrative and may not accurately represent certain medical practices.

Referring to FIG. 2 , a pictorial view of a surgical tool of a surgicalrobotic system is shown in accordance with an embodiment. The surgicalrobotic system 1 includes at least one surgical robotic arm 4. Asurgical robotic arm 4 may be coupled to the surgical robotic platform 5at a mounting location 202, and the surgical tool 7 may be mounted onthe surgical robotic arm 4, e.g., at the distal end of the arm. Thesurgical tool 7 can include a tool driver and a cannula coupled to anend of the tool driver. The cannula can receive and guide a surgicalinstrument, e.g., a grasper, a camera, etc. Furthermore, the robotic arm4 may include several links that can be actuated to position and orientthe surgical tool 7 in a particular manner. More particularly, the linkscan be actuated to cause movement of the surgical tool 7 that emulatesmovement of the UID 14.

Referring to FIG. 3 , a pictorial view of a UID of a surgical roboticsystem is shown in accordance with an embodiment. As described above, auser, e.g., the remote operator 9, can hold and move the UID 14 tocontrol spatial motion of a corresponding surgical tool 7 of thesurgical robotic system 1. The UID 14 can be an ungrounded UID.Accordingly, the UID 14 can have no connection to a supporting structureother than a user's hand, and is capable of being supported and movedfreely in space by the user's hand. It will be appreciated, however,that grounded systems having UIDs connected to a supporting structure,e.g., to a support linkage, may also benefit from the methods ofgrip-dependent control described below.

The UID 14 can include a tracking sensor 302. The tracking sensor 302may be a six-degree-of-freedom electromagnetic tracker that is used togenerate a spatial state signal, e.g., an input pose signal, in responseto movement of a device body 304. More particularly, the tracking sensor302 may be any sensor configured to track movement of the UID 14 in sixdegrees of freedom. The spatial state signal can be used by one or moreprocessors to generate control signals to control proportional motion ofthe arm actuator 17 and/or the surgical tool 7 of the surgical roboticsystem 1. Accordingly, the UID 14 can be used to control highlydexterous, precise movement of the robotic actuator 17 and/or therobotic surgical tool 7. Furthermore, in addition to detecting anorientation of the UID 14, tracking data from the tracking sensor 202may be used to determine a location, speed, or acceleration of the UID14.

The UID 14 can include a proximal end 306 and a distal end 308, andseveral grip components 310 can extend distally from the proximal end306 toward the distal end 308. The grip components 310 can be gripped bya user and manipulated to generate spatial state signals. For example,the user can rotate, tilt, or translate the UID 14 in free space tocause the control system to generate control commands to cause acorresponding end effector of the surgical tool 7 to move similarly.Similarly, the user can press on the grip components 310 to generate agrip signal for controlling jaws of the corresponding end effector.

The UID 14 may also include a device head 312. The device head 312 maybe located at the distal end 308 of the UID 14. In an embodiment, thedevice head 312 has a surface extending transverse to a central axis 314of the UID 14. The transverse surface may be distal to the gripcomponents 310, and thus, may be axially aligned with the user's handwhen the user is gripping the grip component 310 (FIGS. 4-5 ).

In an embodiment, the UID 14 includes one or more proximity sensors 316.For example, several proximity sensors 316 can be mounted on the devicebody 304, the device head 312, or another portion of the UID 14, anddirected toward the user's hand when the user is gripping the UID 14.The proximity sensors 316 may be mounted on the transverse surface ofthe device head 312 and the sensing path of the sensors can be directedproximally in a direction of the central axis 314 of the UID 14 todetect the fingers of the user when the user is gripping the gripcomponents 310. For example, the UID 14 can include three or more, e.g.,six, proximity sensors 316 evenly distributed around the central axis314 of the device body 304 on the device head 312. The proximity sensors316 may be distributed such that the sensor group is able to detect thepresence and/or position of the fingers at any location around thedevice body 304. More particularly, the proximity sensors 316 can detecta target within an annular target zone extending around the device body304. Accordingly, in addition to receiving tracking data from thetracking sensor 302 indicative of a location or movement of the UID 14,the surgical robotic system 1 can receive proximity data from theproximity sensors 316 indicative of whether the UID 14 is being held bythe user, or indicative of whether an object is adjacent to or near theproximity sensors 316.

The proximity data from the proximity sensors 316 may be used to providedrop detection for the surgical robotic system 1. Proximity data fromthe proximity sensors 316 can indicate the presence of the user's handwhen the user is holding the UID 14, and by contrast, can indicate theabsence of the user's hand when the user is not holding the UID 14.Accordingly, the proximity data can indicate a change from the presenceof the user's hand to the absence of the user's hand when the user dropsthe UID 14. In an embodiment, the surgical robotic system 1 can haltmotion of the surgical tool 7 in response to detecting the dropcondition based on the proximity data. Such drop detection and controlof the surgical tool 7 can provide a safety feature to avoid unintendedmovements of the surgical tool 7 that could harm the patient 102.

In an embodiment, the proximity sensors 316 include sensors to detecttouch on an outer surface of the grip components 310. For example, eachgrip component 310 can include a proximity sensor 316 to detect alocation that the user's finger is placed on the outer surface. Theproximity sensors 316 of the grip components 310 may be used instead of,or in combination with, the proximity sensors 316 mounted on the devicehead 312. Accordingly, the UID 14 can have proximity sensors 316distributed around the central axis 314 of the device body 304 to detectthe presence of the user's fingers at any location on the outer surfaceof the device body 304.

The proximity sensors 316 can include sensors that detect the presenceof nearby objects without physically contacting the objects. A varietyof proximity sensor types exist that may be incorporated into the UID 14for such purpose. For example, the proximity sensors 316 can becapacitive proximity sensors, photoelectric sensors, or other types ofproximity sensors. In an embodiment, the proximity sensors 316 includetime-of-flight (TOF) sensors. The TOF sensors may be mounted on thedevice head 312 and can be used for range imaging. For example, the TOFsensors can emit a beam of electromagnetic radiation, e.g., an infraredlaser beam emitted by an infrared laser source of the sensor, andmeasure a return signal reflected by a nearby object to determine apresence of or a distance to the nearby object. Accordingly, theproximity sensors 316 can be TOF sensors to generate proximity dataindicative of whether an object is adjacent to or near the proximitysensors 316. More particularly, the proximity data can be TOF dataoutput by TOF sensors indicative of a distance from the TOF sensors tothe fingers of the user. A position of the fingers on the UID 14 can bedetermined based on the TOF data.

The proximity sensors 316 may include capacitive sensors mounted on orwithin the grip components 310. For example, the capacitive sensors mayoverlay the outer surface of the device body 304. The capacitive sensorsmay sense touch, e.g., a presence of the user's fingers on the outersurface. Accordingly, the proximity sensors 316 can be capacitivesensors to generate proximity data indicative of whether and how theuser is holding the UID 14. More particularly, the proximity data can becapacitive sensor data output by capacitive sensors indicative of auser's grip on the UID 14. A position of the fingers on the UID 14 canbe determined based on the capacitive sensor data.

Referring to FIG. 4 , a pictorial view of a UID being held with a firstgrip configuration is shown in accordance with an embodiment. A gripconfiguration of a user's hand 402 on the UID 14 can be detected by theproximity sensors 316. For example, TOF sensors mounted on the devicehead 312 can measure a distance from the sensor to the fingers of theuser. Based on the distance, a position of each of the fingers on theouter surface of the device body 304 may be determined. These positions,e.g., contact points 403 between the fingertips of the user and theouter surface, can define the grip configuration. For example, in thegrip configuration shown in FIG. 4 , the user's fingers can grip thedevice body 304 radially around the tracking sensor 302. Although thegrip configuration is shown as being detected by the TOF sensors on thedevice head 312, it will be appreciated that the grip configuration canbe detected by capacitive sensors mounted on the outer surface of theUID 14.

The UID 14 can have a center of rotation 404. By default, the center ofrotation 404 can be defined as a center of the tracking sensor 302 thatis used to determine the six-degree-of-freedom position and orientationof the UID 14. For example, the tracking sensor 302 can have a sensorcenter of rotation 406 about which rotation of the tracking sensor 302is tracked. The sensor center of rotation 406 may by default coincidewith a center of mass of the device body 304. For example, the trackingsensor 302 may be integrated in the UID 14 such that the sensor centerof rotation 406 is located at the center of mass of the UID 14. Thedefault center of rotation 404 may be so-positioned because it may beassumed that the user will grip the UID 14 in the first configurationwith the user's fingers near the middle of the device body 304 andradially around, e.g., at a same axial location, as the tracking sensor302. The user may, however, grip the UID 14 at an axial location offsetfrom the sensor center (FIG. 5 ). For example, the user may hold thecontroller in multiple grip configurations as the user changes the gripwhen manipulating the handheld UID 14 during surgery. Each time the gripis changed, the user may expect the center of rotation 404 to changebecause the intuitive user expectation is that the center of rotation404 will be centered within the contact points 403 along the centralaxis 314. The center of rotation 404 may be adjusted to match the userexpectation by moving the virtual center of rotation 404 to a locationoffset from the sensor center of rotation 406.

Referring to FIG. 5 , a pictorial view of a UID being held with a secondgrip configuration is shown in accordance with an embodiment. In thesecond configuration, the user's fingers may be moved backward, orproximally, in the direction of the central axis 314. The contact points403 may therefore be behind (posterior to) the sensor center of rotation406 as the user changes the grip position. In an embodiment, as the gripconfiguration changes, the expected center of rotation 404 changes. Moreparticularly, the user may expect the center of rotation 404 to beradially inward from her fingertips regardless of where the user gripsthe device body 304. Accordingly, when the user has the second gripconfiguration, the user may expect the center of rotation 404 to beposterior to the sensor center of rotation 406 of the tracking sensor302. Similarly, when the user has a third grip configuration (not shown)with her fingertips in front of (anterior to) the sensor center ofrotation 406, the user may expect the center of rotation 404 to beanterior to the sensor center of rotation 406. In any case, the user mayexpect the center of rotation 404 to be at a location along the centralaxis 314 that does not coincide with the sensor center of rotation 406.

Methods of controlling a component of the surgical robotic system 1based on the user's grip on the UID 14 are described below. The methodcan include grip-dependent active or passive control of the component.For example, FIG. 6 includes a flowchart of a method of actuating thesurgical tool 7 of the surgical robotic system 1 based on the center ofrotation 404 of the UID 14 (active control). FIG. 9 includes a flowchartof a method of actuating the UID 14 based on the center of rotation 404(passive control). FIG. 6 is referred to first in combination with FIGS.7-8 , which illustrate certain operations of the methods. It will beappreciated, however, that the methods are specific embodiments of ageneralized method of grip-dependent control of a component of thesurgical robotic system 1, and certain operations can apply to any ofthe methods described herein.

Referring to FIG. 6 , for passive controllers, modifying the center ofrotation 404 to coincide with an expectation of the user can provideoptimal mapping between an input by the user, e.g., a movement of theUID 14, and output orientations, e.g., movement of the surgical tool 7.To ensure optimal control using the UID 14, a virtual center of rotation404 may be determined and used by the control system to control movementof the surgical tool 7 based on the UID 14 movements. The virtual centerof rotation 404 can be a non-default center of rotation 404 that, ratherthan being based on the fixed sensor center of rotation 406, may bebased on the instantaneous grip configuration of the user's hand 402 onthe UID 14.

At operation 602, as a precursor to determining the virtual center ofrotation 404, the surgical robotic system 1 can determine a gripconfiguration of the user's hand 402 on the UID 14. The determinationcan include determining how the user is holding the UID 14 based on theproximity data. Such determination may include detecting, by one or moreof the proximity sensors 316, positions of one or more fingers of theuser's hand 402 on the UID 14. For example, the proximity sensors 316can detect the contact points 403 between the fingers and the outersurface of the device body 304. The contact points 403 can be determinedbased on TOF data from TOF sensors mounted on the device head 312, orbased on capacitive sensor data from capacitive sensors mounted on thegrip components 310. The contact points 403 can define the gripconfiguration by defining the locations on the outer surface of the UID14 at which the user's hand grips the UID 14. Accordingly, determiningthe grip configuration may include determining a spatial coordinate ofeach contact point 403 of the user's grip.

Determining the grip configuration may include determining a relativeposition between the contact points 403 of the user's grip and areference point. For example, the reference point may be a center ofmass of the UID 14, a center of inertia of the UID 14, or the defaultcenter of rotation 406. In an embodiment, the determination includesdetermining whether the positions of the one or more fingers of theuser's hand 402 are anterior to or posterior to the sensor center ofrotation 406 of the tracking sensor 302. This may include, for example,comparing a media or average axial location of the contact points 403 toan axial location of the reference point, and determining whether theuser is gripping the UID 14 in front of or behind the reference pointbased on the comparison. Other techniques of defining the gripconfiguration may be used, as described below, and the describedtechniques are provided by way of example and not limitation.

Referring to FIG. 7 , a schematic view of a grip configuration on a UIDis shown in accordance with an embodiment. The grip configuration can bedefined by the contact points 403. For example, the contact points 403can define a plane 702, and an axial location of the plane 702 relativeto the sensor center of rotation 406 may be used to determine the gripconfiguration. The user grip may include two contact points 403,corresponding to two fingers gripping the device body 304, and the plane702 can be a plane 702 containing the contact points 403 and orthogonalto the central axis 314. Alternatively, the user's grip may includethree or more contact points 403, corresponding to three or more fingersgripping the device body 304, and the plane 702 may be a plane 702containing three of the contact points 403. These are examples, andother manners of determining the grip plane 702 as defined by thecontact points 403 may be contemplated by one skilled in the art. In anycase, once determined, the grip plane 702 may be compared to the sensorcenter of rotation 406 (or another reference point of the UID 14) todefine the grip configuration. When the plane 702 is anterior to thesensor center of rotation 406, the grip configuration can be an anteriorgrip. When the plane 702 is posterior to the sensor center of rotation406, the grip configuration can be a posterior grip. Of course, theplane 702 may intersect (or be within a predetermined distance of) thesensor center of rotation 406, in which case the grip configuration maybe a medial grip.

At operation 604, the center of rotation 404 is determined based on thegrip configuration. In any of the grip configurations describe above(the anterior grip, the medial grip, or the posterior grip), the centerof rotation 404 can be estimated according to predetermined estimationrules. Estimation may not require extreme precision. For example,comparison of the grip plane 702 to the sensor center of rotation 406may lead to the determination that the user is holding the UID 14 withthe anterior grip. In response, the virtual center of rotation 404 maybe set to a point on the central axis 314 midway between the sensorcenter of rotation 406 and the distal end 308 of the UID 14. Similarly,determination that the user is holding the UID 14 with the posteriorgrip may cause the virtual center of rotation 404 to be set to a pointon the central axis 314 midway between the sensor center of rotation 406and the proximal end 306 of the UID 14. The estimated virtual center ofrotation 404 may not be exactly where the user expects the center ofrotation 404 to be, but may be close enough to provide accurate andintuitive control of the surgical tool 7.

Still referring to FIG. 7 , greater precision in the estimated virtualcenter of rotation 404 may be achieved using the contact points 403 asan input to determine the center of rotation 404. The user mayintuitively expect the virtual center of rotation 404 to be radiallybetween the contact points 403 of the fingers on the UID 14.Accordingly, the determination of the center of rotation 404 can includedetermining that the center of rotation 404 is at a location between thepositions of the one or more fingers of the user's hand 402 on the UID14. For example, the center of rotation 404 can be set to the locationthat is radially between the user's fingers, e.g., at a point within theUID 14.

In an embodiment, the virtual center of rotation 404 can be definedbased on a geometric relationship between the grip plane 702 and anotherreference geometry of the UID 14. For example, the virtual center ofrotation 404 can be determined to be a point along the central axis 314at which the grip plane 702, as defined by the contact points 403,intersects the central axis 314.

In any case in which the user's grip is offset from a middle of the UID14, the expected center of rotation may not coincide with the sensorcenter of rotation 406. The sensor center of rotation 406 can be thedefault center of rotation located at a datum of the coordinate systemgenerated by the tracking sensor 302. Accordingly, the sensor center ofrotation 406 can be fixed. That is, the sensor center of rotation 406may be set during manufacturing or during calibration of the system, andonce defined, may remain fixed throughout a surgical procedure. Bycontrast, the expected center of rotation can continuously orintermittently change during use of the UID 14. Thus, the expectedcenter of rotation can be offset from the sensor center of rotation 406by different distances throughout the surgery. Accordingly, the virtualcenter of rotation 404 can be set to a location offset from the defaultcenter of rotation 406 based on the user's grip.

In an embodiment, the determined center of rotation 404 is used togenerate control commands to actuate a component of the surgical roboticsystem 1. For example, in the passive control embodiment of FIG. 6 , thecomponent can be the surgical tool 7 and movements of the surgical tool7 can be mapped to input movements of the UID 14 about the virtualcenter of rotation 404. As described below with respect to the activecontrol embodiment of FIG. 9 , however, the component can be the UID 14and haptic feedback provided by the UID 14 can be mapped to input loadsapplied to the surgical tool 7. In both embodiments, the mapping of theinput to the output is adjusted based on how the user is holding the UID14, e.g., with the anterior grip or the posterior grip.

Still referring to FIG. 6 , at operation 606, in the passive controlscenario, tracking data is received from the tracking sensor 302 of theUID 14. The tracking data is the input data used by the system tocontrol how the end effector of the surgical tool 7 will respond. Moreparticularly, the tracking data corresponds to movement of the trackingsensor 302 in six degrees of freedom, and represents the intendedmovement of the surgical tool 7 in six degrees of freedom.

At operation 608, the surgical robotic system 1 generates, based on thecenter of rotation 404 of the UID 14 and the tracking data from thetracking sensor 302, control commands to move the output component ofthe surgical robotic system 1. The component may be the surgical tool 7of the surgical robotic system 1 such that generating the controlcommands is to actuate the surgical tool 7 based on the center ofrotation 404 and the tracking data to move the surgical tool 7 in sixdegrees of freedom.

Referring to FIG. 8 , a schematic view of a movement of a UID about acenter of rotation is shown in accordance with an embodiment. When thecenter of rotation 404 changes, e.g., based on the user changing thegrip, the mapping between the input movements of the UID 14 and theoutput movements of the surgical tool 7 can be changed to cause theoutput movements to match the expectations of the user.

The expectations of the user depend on the user's grip. The user may beholding the UID 14 with the contact points 403 on the outer surfaceproximal to the sensor center of rotation 406. The user can impart aninput movement 802 of pure rotation while holding the UID 14 near theproximal end 306. The user may therefore anticipate that the center orrotation of the surgical tool 7, e.g., a point along the cannula, willbe controlled to also rotate with pure rotation. It will be appreciated,however, that pure rotation about the expected center of rotation 404,e.g., from orientation COR to COR′, will instead cause a translation ofthe fixed center of rotation 404 from point SC to point SC′. In otherwords, if the center of rotation 404 used to control surgical toolmovement is not adjusted, the point along the cannula will translatethrough space rather than experience pure rotation in space, asintended.

To avoid unintended movements of the surgical tool 7 and to cause thesurgical tool movements to match the expectations of the user, thevirtual center of rotation 404 can be adjusted from the fixed sensorcenter of rotation 406 to the center of rotation 404, and the input canbe mapped to the output accordingly. More particularly, tracking datafrom the tracking sensor 302 can be mapped to movements about thevirtual center of rotation 404 rather than about the sensor center ofrotation 406. Thus, for example, when the tracking sensor 302experiences translational movement from point SC to point SC′, thesystem can map the movement to pure rotation of the surgical tool 7about the tool center of rotation. Accordingly, the generated controlcommands cause movement of the surgical tool 7 that match theexpectations of the user based on the user's handling of the UID 14.

Referring to FIG. 9 , a flowchart of a method of actuating a UID of asurgical robotic system based on a center of rotation of the UID isshown in accordance with an embodiment. As noted above, the component ofthe surgical robotic system 1 for which control commands are generatedcan be the UID 14. For active controllers, such as haptic UIDs 14 withactive torque degrees of freedom, changing the center of rotation 404can be used in order to change the mapping to the surgical toolmovement, as described above, and also to render different torques tothe user holding the UID 14. More particularly, the UID 14 may beconfigured to render haptic feedback to the user based on input loads,e.g., reaction loads, applied to the surgical tool 7, and the behaviorof the UID 14, e.g., the point at which the force is rendered to theuser, can be adjusted to ensure that the user experiences a realisticemulation of the reaction load.

Operations 902 and 904 can be similar or identical to operations 602 and604 described above. Accordingly, prior to remapping the input, such asa reaction load applied to the surgical tool 7 to a haptic feedbackforce generated at the UID 14, the surgical robotic system 1 candetermine the center of rotation 404 of the UID 14 based on a gripconfiguration of the user's hand 402 on the UID 14.

At operation 906, the surgical robotic system 1 receives load data froma load sensor of the surgical robotic system 1. Referring again to FIG.2 , a load sensor 204 can be coupled to a joint of the robotic arm 4, ora portion of the surgical tool 7. The load sensor 204 can measure areaction load or torque associated with a force 206 applied to therobotic arm 4 and/or surgical tool 7. For example, the load sensor 204can generate load data corresponding to the force 206 applied to thesurgical tool 7 when the surgical tool 7 is located within the patient102. The force 206 can be a reaction force applied by the tissue of thepatient 102 as the surgical tool 7 is moved within the patient 102.Accordingly, it will be appreciated that the force 206 may be applied ata location offset from the center of rotation of the surgical tool 7,and thus, would create a force and/or torque about the center ofrotation of the surgical tool 7.

Referring again to FIG. 9 , at operation 908, the surgical roboticsystem 1 generates control commands based on the center of rotation 404of the UID 14 and the load data to actuate the UID 14 to render hapticfeedback corresponding to the force 206. The control commands can renderthe haptic feedback, e.g., a haptic force or a haptic torque, to theuser's hand 402 such that the user feels the reaction force 206 as ifthe user were holding the surgical tool 7 at the surgical site. Thetransformation of the reaction force 206 at the surgical tool 7 to theUID 14 can include mapping the reaction force 206 to the virtual centerof rotation 404 rather than the sensor center of rotation 406. UsingFIG. 8 as an example, if the reaction load is a reaction torqueincluding pure rotation at the center of rotation of the surgical tool7, haptic motors within the UID 14 can be actuated to cause a purerotation about the virtual center of rotation 404 (accompanied bytranslational motion of the sensor center of rotation 406). Accordingly,the user can experience realistic haptic feedback corresponding to theloads seen by the surgical tool 7.

Referring to FIG. 10 , a block diagram of exemplary hardware componentsof a surgical robotic system is shown in accordance with an embodiment.The exemplary surgical robotic system 1 may include the user console 2,a surgical robot 1002, and the control tower 3. The hardware componentsof the surgical robotic system 1 can include one or more processorsconfigured to perform the methods described above. More particularly,the one or more processors, e.g., of the user console 2, may executeinstructions stored on a non-transitory computer-readable medium tocause the surgical robotic system 1 to perform the methods describedabove. Furthermore, the surgical robotic system 1 may include otheradditional hardware components; thus, the diagram is provided by way ofexample and not limitation to the system architecture.

As described above, the user console 2 comprises console computersystems 16 and one or more UIDs 14. User console 2 can include consoleactuators 1004, displays 15, a UID tracker 1006, foot pedals 13, and anetwork interface 1008. A user or surgeon sitting at the user console 2can perform robot-assisted surgeries by controlling the surgical robot1002 using the one or more UIDs 14 and foot pedals 13. Positions andorientations of the UIDs 14 are continuously tracked by the UID tracker1006, and status changes are recorded by the console computers 16 asuser input and dispatched to the control tower 3 via the networkinterface 1008. The tracking data from the UID tracker 1006 and theproximity data from the UIDs 14 can be used by one or more processors ofthe console computers 16 to perform drop detection or grip-dependentcontrol, as described above. Real-time surgical video of patientanatomy, instrumentation, and relevant software apps can be presented tothe user on the high resolution 3-D displays 15 including open orimmersive displays.

Unlike other existing surgical robotic systems, the user console 2disclosed herein may be communicatively coupled to the control tower 3over a single fiber optic cable. The control tower 3 can be a mobilepoint-of-care cart housing touchscreen displays, computers that controlthe surgeon's robotically-assisted manipulation of instruments, safetysystems, graphical user interface (GUI), light source, and video andgraphics computers. As shown in FIG. 10 , the control tower 3 maycomprise central computers 1010 including at least a visualizationcomputer, a control computer, and an auxiliary computer, variousdisplays 1012 including a team display and a nurse display, and anetwork interface 1014 coupling the control tower 3 to both the userconsole 2 and the surgical robot 1002. The control tower 3 may alsohouse third-party devices, such as an advanced light engine 1016, anelectrosurgical generator unit (ESU) 1018, and insufflator and CO₂ tanks1020. The control tower 3 may offer additional features for userconvenience, such as the nurse display touchscreen, soft power andE-hold buttons, user-facing USB for video and still images, andelectronic caster control interface. The auxiliary computer may also runa real-time Linux, providing logging/monitoring and interacting withcloud-based web services.

The surgical robot 1002 comprises an articulated operating table 5 witha plurality of integrated arms 4 that can be positioned over the targetpatient anatomy. A suite of compatible tools 7 can be attached to ordetached from the distal ends of the arms 4, enabling the surgeon toperform various surgical procedures. The surgical robot 1002 may alsocomprise control interface 1022 for manual control of the arms 4, table5, and tools 7. The control interface can include items such as, but notlimited to, remote controls, buttons, panels, and touchscreens. Otheraccessories such as trocars (sleeves, seal cartridge, and obturators)and drapes may also be needed to perform procedures with the system. Insome variations the plurality of arms 4 include forearms mounted on bothsides of the operating table 5, with two arms on each side. For certainsurgical procedures, an arm mounted on one side of the table can bepositioned on the other side of the table by stretching out and crossingover under the table and arms mounted on the other side, resulting in atotal of three arms positioned on the same side of the table 5. Thesurgical robot 1002 can also comprise table computers 1024 and a networkinterface 1026, which can place the surgical robot 1002 in communicationwith the control tower 3.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope of the invention as set forth in thefollowing claims. The specification and drawings are, accordingly, to beregarded in an illustrative sense rather than a restrictive sense.

What is claimed is:
 1. A method of controlling a surgical robotic system based on a user's grip on a user interface device, the method comprising: determining a grip configuration of a user's hand on a user interface device of a surgical robotic system; determining, based on the grip configuration, a virtual center of rotation of the user interface device; and generating, based on the virtual center of rotation of the user interface device, control commands to actuate a component of the surgical robotic system.
 2. The method of claim 1, wherein determining the grip configuration includes detecting, by one or more proximity sensors, positions of one or more fingers of the user's hand on the user interface device.
 3. The method of claim 2, wherein the user interface device includes a tracking sensor having a sensor center of rotation, and wherein the virtual center of rotation does not coincide with the sensor center of rotation.
 4. The method of claim 3, wherein determining the grip configuration includes determining whether the positions of the one or more fingers of the user's hand is anterior to or posterior to the sensor center of rotation of the tracking sensor.
 5. The method of claim 2, wherein the virtual center of rotation is determined to be at a location between the positions of the one or more fingers of the user's hand on the user interface device.
 6. The method of claim 2, wherein the one or more proximity sensors are time-of-flight sensors.
 7. The method of claim 1 further comprising: receiving tracking data from a tracking sensor of the user interface device, wherein the tracking data corresponds to movement of the tracking sensor in six degrees of freedom; and wherein the component is a surgical tool of the surgical robotic system such that generating the control commands is to actuate the surgical tool based on the virtual center of rotation and the tracking data to move the surgical tool in six degrees of freedom.
 8. The method of claim 1 further comprising: receiving load data from a load sensor of the surgical robotic system, wherein the load data corresponds to a force applied to a surgical tool of the surgical robotic system; and wherein the component is the user interface device such that generating the control commands is to actuate the user interface device based on the virtual center of rotation and the load data to render haptic feedback corresponding to the force to the user's hand.
 9. A surgical robotic system, comprising: a surgical tool mounted on a surgical robotic arm; a user interface device; and one or more processors configured to: determine a grip configuration of a user's hand on the user interface device, determine, based on the grip configuration, a virtual center of rotation of the user interface device, and generate, based on the virtual center of rotation of the user interface device, control commands to actuate one or more of the surgical tool or the user interface device.
 10. The surgical robotic system of claim 9, wherein determining the grip configuration includes detecting, by one or more proximity sensors, positions of one or more fingers of the user's hand on the user interface device.
 11. The surgical robotic system of claim 10, wherein the user interface device includes a tracking sensor having a sensor center of rotation, and wherein the virtual center of rotation does not coincide with the sensor center of rotation.
 12. The surgical robotic system of claim 10, wherein the virtual center of rotation is determined to be at a location between the positions of the one or more fingers of the user's hand on the user interface device.
 13. The surgical robotic system of claim 9 further comprising: receiving tracking data from a tracking sensor of the user interface device, wherein the tracking data corresponds to movement of the tracking sensor in six degrees of freedom; and wherein generating the control commands is to actuate the surgical tool based on the virtual center of rotation and the tracking data to move the surgical tool in six degrees of freedom.
 14. The surgical robotic system of claim 9 further comprising: receiving load data from a load sensor of the surgical robotic system, wherein the load data corresponds to a force applied to the surgical tool of the surgical robotic system; and wherein generating the control commands is to actuate the user interface device based on the virtual center of rotation and the load data to render haptic feedback corresponding to the force to the user's hand.
 15. A non-transitory computer-readable medium storing instructions, which when executed by one or more processors of a surgical robotic system, cause the surgical robotic system to perform a method comprising: detecting, by one or more proximity sensors, positions of one or more fingers of a user's hand on a user interface device; determining, based on the positions of the one or more fingers, a virtual center of rotation of the user interface device; and generating, based on the virtual center of rotation of the user interface device, control commands to actuate a component of the surgical robotic system.
 16. The non-transitory computer-readable medium of claim 15, wherein the user interface device includes a tracking sensor having a sensor center of rotation, and wherein the virtual center of rotation does not coincide with the sensor center of rotation.
 17. The non-transitory computer-readable medium of claim 15, wherein the virtual center of rotation is determined to be at a location between the positions of the one or more fingers of the user's hand on the user interface device.
 18. The non-transitory computer-readable medium of claim 15 storing instructions to cause the surgical robotic system to perform the method further comprising: receiving tracking data from a tracking sensor of the user interface device, wherein the tracking data corresponds to movement of the tracking sensor in six degrees of freedom; and wherein the component is a surgical tool of the surgical robotic system such that generating the control commands is to actuate the surgical tool based on the virtual center of rotation and the tracking data to move the surgical tool in six degrees of freedom.
 19. The non-transitory computer-readable medium of claim 15 storing instructions to cause the surgical robotic system to perform the method further comprising: receiving load data from a load sensor of the surgical robotic system, wherein the load data corresponds to a force applied to a surgical tool of the surgical robotic system; and wherein the component is the user interface device such that generating the control commands is to actuate the user interface device based on the virtual center of rotation and the load data to render haptic feedback corresponding to the force to the user's hand.
 20. The non-transitory computer-readable medium of claim 15, wherein the one or more proximity sensors are time-of-flight sensors. 